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1. EXECUTIVE SUMMARY 


Although technology is front-and-center in today’s military operational environment, people are the 
foundation of our strength. An assessment on the Future Operational Environment! by the U.S. Army 
Training and Doctrine Command (TRADOC G-2) through 2050 underscores the rapid societal changes 
spurred by breakneck advances in Science and Technology (S&T) and how these changes will impact the 
art of warfare. Virtually every S&T advance intersects with other technologies, thus increasing the speed 
of innovation and adoption across the entire S&T portfolio. This convergence allows our adversaries to 
close the technical gap our military has typically enjoyed and increases the speed of human interaction and 
cognition required to successfully execute the mission. The human element is paramount. The ability of our 
personnel to maintain a higher level of creativity, problem-solving, and out-of-the-box thinking is our 
greatest asset and our biggest differentiator. Therefore, the way we educate and train our personnel directly 
impacts our future readiness. 


Military education and training encompass many different schools, universities, and training programs 
designed to foster technical, professional, and leadership skills in military service members. Historically, 
there has been a separation between the education and training communities across the services. Education 
occurs incrementally and involves grappling with ambiguity while thinking and reflecting about the 
concepts being learned’. Training is linked to readiness and offers opportunities to apply an individual’s 
knowledge, skills, and abilities in a manner that provides immediate feedback and progress measurement. 
Within the current context, training and education also have different reporting structures, motivations, and 
logistical requirements such as fuel, personnel, and the access to the appropriate environments or 
equipment. 


Future warfighters must rapidly prepare and adapt to function in an increasingly volatile and complex 
environment. Where current assessment methods judge knowledge retention for education and skill 
proficiency for training, there is a significant gap between those metrics and operational readiness and 
performance. Future force management and readiness metrics must directly link preparations to operations. 
The class of metrics used to do this is referred to as competencies — a measure of how well someone can 
use their acquired skills and knowledge to complete task(s). To enable the future learning ecosystems, all 
the ways to assess the ability of a person will need a digital representation. The on-going research on 
Experience API (xAPI) provides the capability to capture any transaction of learning. The Advanced 
Distributed Learning Initiative (ADL) is continuing to explore how those xAPI transactions can be trusted 
as evidence from an authoritative source. 


Competency Management Systems (CMS) are the brokers of trust. That trust is based on the validity of 
evidence, which consists of pre-existing competency frameworks that capture the requirements of a role 
and is then aligned to frameworks to generate assertions. The ADL Initiative has identified a series of 
technical capabilities that current learning capabilities will integrate in the future to become variants of the 
Total Learning Architecture (TLA). The ADL Initiative has invested in CMS prototypes and instantiated a 


' TRADOC G-2, “The Operational Environment and the Changing Character of Future Warfare”, Army Capability 
Integration Center (ARCIC), Campaign of Learning Technical Report http://www.arcic.army.mil/App _ 
Documents/The-Operational-Environment-and-the-Changing-Character-of-Future-Warfare.pdf 

? J, Johnson-Freese, “The Reform of Military Education: Twenty-Five Years Later” Orbis, Volume 56, Number 1, 
Winter 2012, http://www.dtic.mil/dtic/tr/fulltext/u2/a570086.pdf 


reference implementation in fiscal year 2018 (FY-18) within a controlled environment. In FY-19 The ADL 
Initiative will expand its testing of Competency Based Learning (CBL) and its role within the TLA through 
multiple research projects with the intent for transitioning a capability for Government-wide use. The 
projects include partnering with the Air Education and Training Command (AETC) to validate CMS within 
their environment. That effort will involve multiple industry partners and support from a DoD lab. 


This report presents an early glimpse into the technical complexities of migrating the DoD towards a 
competency based educational system in support of the next generation of talent management / talent 
development to maximize readiness. The report will also provide information on the Competency & 
Credential work to date by the ADL Initiative, and the work expected in FY-19. Appendix A includes 
information about concepts and definitions of CBL. It is advised for readers unfamiliar with CBL to review 
the last section before reading the total document. 


2. INTRODUCTION 


Personnel readiness to execute missions is at the center of Department of Defense’s (DoD) human capital 
management strategy. Under current practices, the Uniformed Services provide enough training and 
education (T&E) for personnel to reach a minimum level of proficiency. Credentials are used to codify 
proficiency for detailing and evaluating personnel. In many cases, these credentials do not expire. However, 
in high reliability and regulated communities such as nuclear power, healthcare, or aviation, outside 
accrediting authorities require on-going demonstration of individuals’ proficiency. 


Within the DoD, entry-level initial training includes formal schools, eLearning courses, and some practical 
lab work. In the field, service members primarily learn from each other and their own experiences, 
augmented by performance support and on the job training to maintain their proficiency. Studies show that 
70-90% of actual learning happens on the job. However, the methods for capturing individual skills and 
their contribution to mission success are disconnected and only documented by freeform narratives in 
evaluation reports. The lack of well-defined performance indicators and performance metrics (competency 
models), the lack of a fine-grained view into how credentials apply to the work environment, and the lack 
of ability to correlate/coordinate key performance variables across a career all limit the predictive utility of 
current T&E systems’ contribution to readiness. Historically, only after a disaster or a systemic problem 
has occurred is the analysis between readiness and T&E performed. 


CBL encompasses all the knowledge, skills, attitudes, and other aptitudes, abilities, motivations, and traits 
(KSAOs) required of a service member in their operational environments. CBL is analogous to Competency 
Based Education but acknowledges military training as a separate endeavor from education. It provides an 
approach towards collecting and collating performance and learning data such that time spent in formal 
T&E is optimized, and the relationship between learning, human performance, and mission effectiveness 
is made explicit. 


The traditional model of education, born in the industrial age with a one-size-fits-all approach for T&E. It 
places an instructor who utilizes a curriculum developed by Instructional Systems Designers (ISD) to 
transmit instructional information to a student. The student undergoes an assessment to verify receipt and 
processing of information. This receipt and processing of information is subject to individualized tailoring 
by the instructor. If the information loss is minimal (a passing grade is achieved), the student is certified 
and transferred to the work environment. Learning continues on-the-job through their own experiences, the 
mentorship of peers, and their familiarity with the challenges of the work environment. However, if those 
elements are captured at all, it is only in periodic job reviews and personnel evaluations. The curriculum 
may be updated periodically, but a significant operational failure is often necessary to initiate any 
significant change in andragogy and/or heutagogy. The gap between learning (training and education) and 
operational performance induces risk to force readiness and risk to mission. This is depicted in Figure 1. 


Individualized Learning Feedback 


bane N 
P instructional Materiay. 
Curricular BY ag 


Review Learn 5 
Period of Review > ; Ng Assessment 
of Mission Curriculum Instructor Student lA 


Effectiveness 


Figure 1. Traditional Training and Education review and update lifecycle 


The CBL model views human operators and the systems they interact with as a Joint Cognitive System 
(JCS)*, where communication occurs between the human operators and the different systems/components 
(including other operators) they interface with. The requirements for these communication channels are 
codified in the competency frameworks that represent the collection of jobs, tasks, conditions, standards, 
and other relationships that occur between human operators and system components. These communication 
channels are mediated by the usability of each system and the experience of the operator. Analysis of these 
communication channels provide insight into improving the effectiveness of the channels, either by 
improving usability, or by scheduling learning activities to provide just-in-time training through a feedback 
mechanism. Thus, learning activities become part of the improvement process between the human operator 
and the component interface. This relies on the creation of performance indicators and performance 
measurement of operational systems that sit outside of traditional T&E. 


In practice this is a complex web of concepts, as there are many operators, components, and competencies, 
as well as the different types of data that exist along a continuum of time constants. Migrating toward this 
method from the current credential-based model requires normalization of human performance data into a 
trusted and consistently metered data representation. This normalization must, therefore, reduce the 
complexity of the many-to-many relationships between competency, work, JCS components, metrics, and 
human operators. The finer grained data, at a much higher sampling rate, however, enables optimization of 
human performance and concomitant mission effectiveness in a way not currently possible, and only 
performed after something drastic has happened. This is depicted in Figure 2. 
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Figure 2. Operators and Components as a Joint Cognitive System - Jn many of today’s organizations, technology 
has become so pervasive that it functions as a frequent enabler of human performance. By analyzing these channels 
of communication, we enable learning as a Continuous Feedback System. 


3 Woods, D. D. & Hollnagel, E. (2006). Joint cognitive systems: Patterns in cognitive systems engineering. 


3. COMPETENCY BASED LEARNING AND READINESS 


CBL promotes the concept of a competency framework comprised of competency objects that represent the 
relationship among the KSAOs of an individual performing a job. As shown in Figure 3, a competency 
framework is a collection of competencies that provide clarification about successful performance in 
different roles at different levels throughout the organization. A competency framework organizes the 
competencies needed to perform successfully in a work setting, such as a job, occupation, or industry. The 
competency framework can be used as a resource for developing curriculum and selecting training 
materials, identifying licensure and certification requirements, writing job descriptions, recruiting and 
hiring workers, and evaluating employee performance. 
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Figure 3. Competency Frameworks - A competency framework is a representation of the competencies required for 
a certain role within an organization. They are comprised of competencies and sub-competencies that can be shared 
across frameworks. Although shown as a hierarchy above, they are not necessarily linear. 


In the industrial model, KSAOs are substituted with Learning Objectives which are taught in a Program of 
Instruction (POI). The POI typically includes several formative assessments, followed by a summative 
assessment (whether in test form or as a subjective evaluation by a qualified person) to measure learning 
transfer from the course to the student. The system is based on trust in the observers and instructors to 
properly evaluate and capture records. Once the final grades have been tallied, many details are lost. 
Moreover, the completion of the POI is often used to justify the conferral of a credential, after which point, 
the credential is the only record on file.* Accreditation is a key component of trust in the education field’s 
interpretation of competency. Accrediting bodies engender trust through open processes and an incentive 
structure that ties high standards to profit or growth. 


In the CBL model, the competency frameworks establish a clear linkage between individual KSAOs and 
organizational performance (e.g., Talent Management / Talent Development). These frameworks, in turn, 
are built on trust. Assertions against the framework are only as valid as the evidence that supports them. 
The data strategy for every competency must align the observable performance indicators for each of the 


4 Regulations typically require archival of grades to provide transcripts, but this information is neither portable with 
the servicemember, or readily available. 


KSAOs encapsulated into each competency. Some evidence will be derived from T&E systems, while the 
large preponderance of evidence will likely reside in the operational environment. Even though a lot of this 
data already exists, the availability of this data across the DoD enterprise is limited due to legacy technology 
architectures and organizational stovepipes of information. A coordinated data architecture and digital 
transformation process must realign these organizational firewalls to be successful in implementing CBL. 


Digital competencies are the currency of human performance. Therefore, assertions must be based on a 
trusted, evidentiary record that documents the situational context of the interactions taken by the human 
operator, and the state of the systems being interacted with in the workplace. A collection of assertions 
provides an estimation of individuals’ capability to meet mission, which are validated against operational 
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Figure 4. The Path to CBL - Stated simply, evidence of performance from trusted agents provides assertion of 


competency. Competencies evaluated against Operational data inform inferences of the relationships (weights) 
between competency and readiness, such that, over time, we can measure the competencies of an organization and 
make reliable predictions of its state of readiness. 


Over the past decade, the DoD has made rebuilding the readiness of the military force one of their priorities 
and has outlined this priority in a key strategic guidance document. However, no comprehensive plan has 
been established to coordinate this effort’. Currently, the policy of blind trust in credentials introduces risk, 
because credentials obfuscate the actual state of a person’s competency. Recent disasters in the 7" Fleet® 
have been attributed to lack of training, inadequate practice while on-board, and lack of sleep among a 
litany of other issues. These causes were driven by cultural failings to accurately report readiness. These 
cultural issues were intensified, since there were no systems in place that shared the actual measurements 
of performance and state of the ships that were not dependent on human interpretation. 


5 Military Readiness GAO-16-841, p.6 https://www.gao. gov/assets/680/679556.pdf 
6 7 Fleet GAO report https://www.gao. gov/assets/700/6959 1 1 .pdf 


Through DoDI 1322.26, the DoD recognizes xAPI as the vehicle for capturing human performance data 
from any system. By extending the capture of data from T&E systems to operational systems, CBL can be 
used to evaluate round-trip estimates of readiness. Figure 4 provides a theoretical path to reaching CBL in 
the DoD. This requires a radical rethinking of the data silos present by our current organization, which is a 
problem common to any digital transformation. By gathering and correlating these data sources, we can 
build towards a predictive system for readiness. This predictive system can also provide early demand 
signals for the overall human capital supply chain management problem of military personnel. 


4. COMPETENCIES IN THE TOTAL LEARNING ARCHITECTURE (TLA) 


Within the context of the TLA, the Activity Stream(s) stored in the Learning Record Store (LRS) provide 
the evidence upon which competency assertions are made. The Competency and Skills System (CaSS) is a 
TLA component that generates rich and traceable data about learning experiences and how they relate to 
skill proficiency, ultimately resulting in a certified set of credentials for the associated KSAOs. Each role 
in an organization is required to perform a set of jobs. Each job has its own set of competencies needed to 
perform the job effectively. By defining these competencies, policy can enable a standardized approach to 
performance that is clear and accessible to everyone within an organization. 


Competency frameworks are expressions of the work environment. Many factors that affect the nature of 
each framework include the native language of its learners, goals of the assessments, availability of software 
engineering support, and the role of accrediting bodies. All these factors drive customization, and more 
importantly creativity in this space. CaSS addresses a reality of CBL where organizations will develop their 
own frameworks and may never converge on a single standard. For example, this is evident in how nuclear 
power personnel and aviation personnel are certified for their work. Even though they work in the same 
service and on the same platform, it is unlikely that all facets of their work can be aligned under one 
competency framework. 


2018 TLA Competency Management Process 


FLUENT: FLUENT uses data from the CaSS Learner Profile (retrieved via an API) to recommend the next activity to the learner via its 
algorithms. FLUENT algorithms also use the learner’s sequence of activities and results (from the LRS), plus the learner’s traits and 


preferences (stored in the CaSS Learner Profile as assertions about purpose-defined frameworks). FLUENT continually updates its views of 
learner traits and transmit them to CaSS as assertions. 
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Figure 5. Competency and Skills System (CASS) — The 2018 TLA Reference Implementation used CASS to create, 
manage, and share information about competencies with other TLA components. 


CaSS ingests multiple different competency frameworks and aligns the different attributes about each 
competency to provide a common language and translation method for defining competencies, the 
associated evidence of attainment, and the different relationships that intertwine with each competency. It 
implements a well-defined set of attributes and rule structures based on Schema.org terms that are 
encapsulated using linked data. CaSS also stores assertions and evidence over time to record the progression 
of a learner pursuing competencies within a framework. Key to establishing assertions of competency from 
an Activity Stream is the reconciliation between the Actor in an xAPI statement (stored in an LRS) and a 
persistent Learner Profile. 


As shown in Figure 5, the 2018 TLA reference implementation, CaSS pulls xAPI statements from the LRS 
and checks the Verb to determine whether the statement claims any evidence of competency. For statements 
that infer competency, the ObjectID from the xAPI statement is compared to the Activity Registry to 


determine which competencies a statement is evidencing. For 2018, all content was treated the same and 
incremental increases in competency were assigned for each completed activity. The 2019 reference 
implementation will include the “weighting” of evidence based on the metadata stored in the Activity 
Registry that describes each activity. However, this approach was not supported by the recommender. 


The primary output from CaSS is a series of Mastery Estimates for an individual over time. This data is 
managed within the competency service layer and written to a machine-readable learner profile that other 
TLA systems can access. The CaSS project also investigated blockchain technologies for managing some 
elements of competencies. A proof of concept was deployed to the Ethereum Blockchain to validate that 
assertions of competency into a Learner Profile from CaSS can be recorded in perpetuity on a Blockchain. 


4.1 Enabling Complex Systems with Competency 


This section provides a theoretical scale for measuring the complexity of a TLA implementation. The five 
TLA adoption levels shown in Figure 6 build upon the basic eLearning and classroom pipelines commonly 
in use today. This scale is focused on architecture and does not prescribe how implementations are 
instantiated. The determinations throughout this paper are based on the critical role CBL plays in maturing 
any learning ecosystem to the point of providing adaptive and intelligent capabilities to its learners. 


Current training and education environments include a wide range of learning activities that are not fully 
connected. Learning management systems (LMS) and other digital technologies (e.g. simulations, hand- 
held learning devices) supplement the student-teacher relationship. Most instruction is linear, composed of 
terminal and enabling learning objectives (or equivalents), and planned in terms of a constrained schedule 
for student attainment of minimum standard to satisfy the award of credentials. 


As a learning system implementation matures to the first TLA adoption level, the introduction of xAPI and 
an LRS increases the richness of analysis and decision support. Activity tracking data (e.g., xAPI) pulls a 
myriad of performance data out of any range of learning activities and aggregates the data for analysis using 
commercially available tools. The xAPI and LRS can normalize classroom data with data collected from 
an LMS, simulators, on-the-job work experiences, or any other network-connected Learning Record 
Provider (LRP) into a data store that is accessible for review. 


TLA adoption level 2 incorporates dashboard visualizations tailored to specific roles within the TLA. 
Instructors, students, instructional designers, administrators, and leaders see performance data and have 
better understanding of past performance when making decisions. This affords instructors the opportunity 
to facilitate interventions based on data from outside traditional classroom or eLearning environments. 
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Figure 6. TLA Services to support Competencies and Credentials - Within the TLA, an activity stream (xAPI) 
captures what the user is doing and which learning activity the learner’s experience is being tracked. The xAPI 
statements record predefined experiences related to learning within each learning activity. These statements are 
stored in an LRS as shown in Level. This empowers detailed analytics based on a learner’s performance across 
numerous LRSs as shown in Level 2. The TLA can also incorporate a competency management capability as shown 
in Level 3. This must be populated with a complete framework that describes the role and requirements of a learner, 
retains enough data about their progression, and includes processes or functions to ensure that included data can be 
trusted. As more data are collected about learners, learning activities, and competencies, the TLA is afforded the 
ability to adapt the sequencing and delivery of different, yet related learning experiences that are tailored to the needs 
of everyone. Eventually, the TLA will tie into other services and systems (e.g., Resource Management Systems) to help 
determine the availability of different learning activities outside the traditional boundaries of Distributed Learning 
(DL). This enables a data architecture that supports lifelong learning. 


The promise of TLA adoption level 3 is using this information to augment the instructional design of 
educational and workplace interventions based on pursuit of competency. Competency management allows 
for additional customization of the learner path, and most importantly, learners are freed from the linear, 
schedule-constrained, course-centric model. Within a CBL system, a learner can benefit from learning 
compression — completing course modules versus entire courses, focusing specifically on the elements of 
competencies they are lacking. 


The TLA adoption level 4 includes additional data richness in the competency and activity-tracking services 
to enable adaptation. Adaptation services provide more tailored experiences for a learner and the ability for 
the TLA to optimize its approaches across a student population. Adaptation includes anything from content 
recommenders to intelligent systems or intelligent tutors. A TLA adoption level 5 fully leverages machine 
learning, providing integrated macro- and micro-adaptation for planning a continuous-learning path, but 
also for optimizing training resources. In practice, these will likely remain manual activities, but the use of 
the integrated data strategy in the lower level implementations should greatly enhance the Electronic 
Decision Support Systems (EDSS) available to aid in those decisions. 


4.2 Authoritative Sources and Authoritative Storage 


Authoritative Sources and Authoritative Storage as formal designations of a given capability are seen in 
DoD network architectures today. Systems or agents that have these titles are critical components of trusted, 
actionable data moving throughout the DoD. Authoritative Sources are considered the trusted reference 
system that generates specific data. Authoritative Storage is the location where a record is held in reference 
to an Authoritative Source. The Naval Information Application Product Suite (NIAPS)’, which is a 
collection of tools that support the Fleet at sea, is a good example of such a record. On-board, the NIAPS 
hosted applications are the authoritative source and storage point for the local Commander. They may have 
information that is more up to date than the record available to the US Navy on shore. This reality does not 
make a NIAPS server on-board authoritative beyond the ship. Those servers are subject to bandwidth and 
operational restrictions that makes their updates unpredictable. The US Navy views the NIAPS servers on 
shore as authoritative source and storage point since they are the most-timely and accurate in respect to all 
ships using NIAPS. 


4.3 How Competency Enables Complex Systems 


At TLA adoption level 3, the presence of a Competency Management System represents a paradigm shift 
within an organization and its learning ecosystem. The use of competency frameworks allow an 
organization to accept non-structured data into their estimation of human performance. To do this, business 
process, instructional design, systems acquisition and, most importantly, culture must change. Instructional 
design within the DoD tends to follow the processes identified in MILHDBK 29612-2A Instructional 
Systems Development/Systems Approach to Training. This handbook enables procuring and maintaining 
Shareable Content Object Reference Model (SCORM) compliant distributed learning. Across DoD, the 
ability to capture learner data equally across all systems does not exist. At level 3, ISDs within the DoD 
will require new standards, processes, and guidance to align their content within more complex systems. 


ISDs will also require processes and tools to build frameworks and align activities within the work 
environment that can generate assertions. The ISDs must then be backed by policy that will allow them to 
ensure these assertions are met within the engineered systems. The current Systems Engineering Technical 
Review (SETR)® processes exist to meet functional requirements for training systems and operational 
systems. SETR must include additional processes and support to look at driving a competency framework 
through human interactions with systems. Functional Requirements for Competency is a new concept to 
SETR but must be addressed. xAPI can capture the activities of Subject Matter Experts (SMEs) and 
learners, but systems will need to be constructed that can leverage that data during their total life-cycle. 


The current Defense acquisition process fails to populate all the data available regarding human 
performance into a common data layer within its networks. Weapon systems process tremendous amounts 
of data and depend on the interaction of dozens of personnel. The potential magnitude of evidence that 
work systems can generate can drive competency frameworks to become true models of human 
performance. To move DoD communities to a TLA adoption level 4, analysis is required to identify the 
barriers blocking the instrumenting of all systems to generate performance data. 


TNIAPS https://www.public.navy.mil/spawar/NA V Y31 1/Pages/NIAPS_Info_Center_FAQs-Basics.html#Q1 
Shttps://www.dau.mil/cop/se/DAU%20Sponsored%20Documents/SETR %20Process%20Handbook %20v 1.0%20201 


50206.pdf 
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4.4 Moving from Level 2 to Level 4 of the TLA 


As envisioned, each school, simulator system, or other collection of learning activities will establish its own 
LRS. The xAPI specification enables federated LRSs, which results in managed authoritative data about 
each learner. Each unique LRS, through permissions, enables a trusted relationship that hold statements 
about an individual’s performance that can roll up into an aggregated learner profile as shown in Figure 7. 
The transactional layer represents all xAPI data. This xAPI data either promulgates to an authoritative LRS 
or remains within local systems. The concept of internal vs external statements is centered on how critical 
the statements are to capturing training records for credentials. Some LRSs and systems may be allowed to 
issue assertions that provide credentials to users. 
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Figure 7. TLA Adoption Level 2: Federation of LRS Supporting a Learner Profile - xAPI statements are captured 
in an LRS that can be associated with any learning activity, or collection of activities. These are federated data sources 
that can be rolled up into different levels of granularity. 


Each of the connected LRSs in Figure 7 collect data at varying levels of granularity. A flight simulator 
may collect fine-grained statements about the learner inside a simulated scenario, while an LRS associated 
with a school may track the student at a much higher granularity (e.g., formative/summative evaluations, 
transcripts). In this capacity, all learning records are preserved in the system in which they are created. 


For a competency management service to be instantiated within the TLA, another LRS must exist that 
collects statements rolled up from the transactional layer. This is due to the Kafka streaming architecture 
set up for the 2019 TLA reference implementation. As shown in Figure 8, each LRS at the learning 
transaction layer will stream statements into a common LRS. The competency management service does 
not care about every statement a learning system makes about the learner. Instead, the CMS is more 
interested in when key events happen that assert something important about the learner. Assertion 
statements are captured in a different federation of LRSs that preserve competency data. Ideally, this 
arrangement would allow legacy systems that do not align to competency frameworks (legacy LMS, 
simulators, and observed human performance) to change through attrition. 
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In the future, all systems used to provide evidence of competency, both T&E and operational, may have an 
LRS, business logic, and authoritative data sources identified. The learner profile service will draw on 
multiple, authoritative sources to present a snapshot what an individual knows at any given point in time. 
It can also act as a source for other systems to base decisions from such as simple adaptive systems and 
scheduling systems. It can act as the authoritative source for multiple authoritative storage points under this 
model. This would afford the DoD a trusted source but allow the services to maintain authority over their 
own data. 
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Figure 8. Adding Competency Management - The addition of a competency management system enables assertions 
against specific competencies held inside any number of competency frameworks. Assertions are made from the 
federated LRS data that collect performance information from T&E and operational systems. 


Figure 8 shows the connection from the transactional LRSs sending statements through the Kafka stream 
to an LRS that aggregates statements from multiple learning activities and feeds them to a CMS. The CMS, 
in turn, infers competence and writes the mastery estimates to the learner profile. In the learning ecosystem, 
legacy authoritative LRSs would not be accessed by the learner profile once this level of maturity is reached. 
The CMS can still capture and trust appropriate legacy asserted xAPI statements but would also scan its 
other contents and compare this to trusted frameworks. At scale, it is important to preserve interoperability 
between LRS federations, competency frameworks, and learner profiles. 
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This level of TLA adoption provides a level of maturity where complex systems can provide the basis for 
applying machine learning to human performance. The T&E and operational systems that have been 
instrumented to report performance can generate a large amount of data that can be analyzed to issue 
credentials, track assertions that indicate skill decay, and capture when managers revoke credentials due to 
behavior. The LRS Service, which captures all authoritative data from activity providers, is the point where 
other TLA components can view evidence. The competency manager leverages that evidence, aligns it to 
a competency framework, and generates trusted assertions in the form of xAPI. That trusted xAPI data is 
then stored in the Trusted LRS. The Trusted LRS provides portability of competency data since it is now 
in a standard format (xAPI) and allows TLAs to change out competency systems as necessary without 


losing their historic records. 
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Figure 9. TLA Architecture with Two LRS Federations - Loosely based on the 2018 TLA reference 
implementation, this graphic shows the inclusion of a recommender that has access to both the learner profile and the 


transactional LRS data store. 


Figure 9 presents a mature version of the future learning ecosystem. This is based on the 2018 TLA 
reference implementation There is a high-likelihood that some systems (e.g., intelligent tutoring systems, 
recommenders, simulators) will need direct access to the transactional layer instead of only accessing the 
authoritative data stored in a learner profile. This instantiation can aggregate learner data from all types of 
systems in a way that allows a recommender to provide just-in-time feedback to an individual based on 


their actual performance. 

The evolution towards this level of TLA adoption requires change across all the institutional barriers across 
the DoD. Changes will be required in the policies and processes we use to acquire systems such that they 
can be instrumented to publish key performance indicators that can be aligned to competencies. Lastly, the 
massive undertaking of creating frameworks, aligning evidence, and gaining access to that evidence is not 


trivial. 
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5. DISTRIBUTED ACYCLIC GRAPHS AS THE EXPRESSION OF DIGITAL COMPETENCIES 


The Institute of Electrical and Electronics Engineering (IEEE) Learning Technology Standards committee 
(LTSC) has defined a Data Model for Reusable Competency Definitions (RCD) under their 1484.20.1-2007 
standard. This is a critical data point for discussion of competency within the TLA standards-oriented 
ecosystem. The IEEE defines an RCD as “any aspect of competence, such as knowledge, skill, attitude, 
ability, or learning objective.” 


Using this as a guide, any formal learning opportunities in use today would preserve the Enabling Learning 
Objectives/Terminal Learning Objectives (ELOs/TLO) structure in place by mapping those ELOs/TLOs to 
one or many RCDs within many frameworks. As shown in Figure 10 below, the nodes within a competency 
framework can have many relationships that are bi-directional in nature. From this perspective, competency 
is not strictly hierarchal, and a single job may align with multiple competencies. Experience has shown that 
many frameworks are structured both in hierarchy and in time. Time represents two variables within CBL: 
the linear progression of how traditional education is scheduled and the rate that competency degrades in 
an individual. 


Mathematically, the RCD is expected to behave as a Distributed Acyclic Graph (DAG). In graph theory, a 
DAG structure consists of nodes that are connected to each other with edges. Each node in the graph is 
analogous to a competency object and the edges define the relationship between them. Each node may have 
numerous edges connected to it and the relationships are directed in that they have a direction. ADB is not 
the same as B>A. Acyclic infers that the relationships are non-circular and that when moving from node 
to node by following the edges, you will never encounter the same node a second time. 


Competency Framework 


Depth of RCDs Representing Mastery — 
Alignment to Content & Media 
—- Formal 


Education 


Informal 
Education 


Figure 10. Relating a Framework to Learning - JEEE 1484.20.] standard for Reusable Competency Definitions 
provides the foundation for a competency framework that can be used to align learning content, formal and informal 
training opportunities, and work experience. 


Attaining mastery is the goal of individuals being managed within a competency framework. RCDs can be 
affected by a variety of assertions under a given set of standards and conditions. Two individuals may 
experience different working conditions for their given role, but the related framework must track mastery 
without any biasing. Most competency frameworks in industry and academia start with a single content 
object and extend to other learning activities to support its pursuit academically. Content objects usually 
include media, courseware, and scheduled on-the-job training. 
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This approach accelerates the obtainment of a credential but does not allow informal learning and work 
experiences to be included in the calculation. Skill decay mapping is also decoupled when the focus is on 
content since the updating of content is highly dependent on funding cycles in the DoD. Content is part of 
logistics? and its updates are driven by two factors: schedule or workplace problems. Competency assertions 
offer the opportunity to drive updates to available formal and informal learning based on actual human 
performance. 


The concept of utilizing performance data from operational systems to predict competency is relatively 
new. This is substantially different than traditional educational pipelines. In those pipelines, assertions are 
scaled to fulfilling a credential, which typically includes numerous competencies. Competencies are 
independent of time and may be completed piecemeal due to the experiences of individuals. The varying 
experiences of individuals will initially require a predetermined structure of their “weight” in terms of the 
overall framework. Weighting for a single competency should reflect its complexity in relation to 
supporting competencies and is highly dependent on the context of how a learner applies this competency 
to their job. The alignment of evidence to competencies is highly contextual as well and will require 
weighting to quantify the different assertions being made. 


A person lacking competency in basic concepts will struggle with more complex tasks. Different learning 
activities infer different levels of competence at different times within the continuum of learning. Thus, the 
weighting becomes a multivariate equation that involves the contextual weighting between related 
competencies, the weighting and currency of evidence, and the weighting of assertions that increase or 
decrease over time based on the context of the evidence. This would also afford competencies with multiple 
weights that reflect their context in a job or task. 


6. THE WAY AHEAD 


A 2016 Institute for Defense Analysis (IDA) report,'° identified six key research areas related to 
competency in practice within an organization. The report’s conclusions are repeated below: 


Credentialing. Credentialing is one accepted means of verifying personnel qualifications that correlate 
with successful employment. In addition, credentialing must address employers’ changing requirements 
and individual workers’ changing skill levels over time. There are numerous credentials, both digital and 
paper-based, that need to be aggregated and accounted for. This is more of an engineering challenge than a 
research challenge. 


Automation. The development and use of competency systems is resource intensive. More efficiency in 
implementing a competency-based system of personnel management is essential to facilitate widespread 
use. Automation is needed to support the generation of metadata, calculate the weighting of evidence, 
measure its context, calculate the roll-up relationships of different competency relationships based on the 
context of the job or role its being applied to, and generate competency frameworks from existing and 
disparate types of text-based documentation already available within the DoD. 


° NAVFAC Logistics guidance, see section 4.7. 
https://www.navfac.navy.mil/content/dam/navfac/Expeditionary/PDFs/NA VFACINST408 1 -1A-Encl1.pdf 

10 TDA, Review and Assessment of Personnel Competencies and Job Description Models and Methods, Belanich J et 
al, 2016 IDA Document D-5803 
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Taxonomy. Diverse structures and categorization taxonomies are used across organizations and 
occupational databases to describe competencies. This leads to difficulties in comparing people or roles. 
Additional research is needed to understand how controlled vocabularies might be used to describe different 
attributes of a competency. 


Granularity. Inconsistency is common in the level of detail used to describe a specific position, role, or 
broader occupation within competency systems. Such variations complicate decisions about which 
competencies to use. Further, different learning activities can generate different amounts of meaningful 
data.. 


Validity. The relevance of competencies to roles and performance is not well substantiated. Content 
validity is the most common form of assessment but is less robust than assessment that depends on 
quantitative performance evidence. 


Tailoring. The process of adapting general competency structures and databases to suit organizational 
missions and functions can take extensive effort and present many difficulties. Organizations can study 
what others have done or extract from generic databases of competencies to leverage existing resources. 
However, even such initial steps are resource intensive. 


In the two years since this report was delivered to the ADL Initiative much of its content is still true. There 
are numerous competency systems in existence, but many barriers to connecting them. The work across 
DoD and civilian sectors to afford interoperability among these data silos is critical. A competency system 
is dependent on the validity of the frameworks, data flows, and evidence in the surrounding environment. 
The system should service, align, and preserve all necessary frameworks it services, rather than translating 
them into a specific language and discarding their previous data. This facilitates traceability over time and 
across frameworks thus enabling records portability. This data also needs to be immutable, validated, 
preserved, and ultimately shared with other learning systems that have a need for it. From an organizational 
perspective, additional research needs to be performed around the intersection of individual and team 
competencies. 


CBL will also benefit from organizational vocabularies and classifications of competencies that can be 
attached as metadata to each competency object. This allows for a controlled vocabulary and descriptors 
that can be aligned across organizations. Metadata vocabularies might include descriptors that inform 
whether a competency includes psycho-motor skills, fine motor skills, cognitive skills, skill decay, or 
relevant environmental factors that affect or inform the description of a competency. This information is 
already collected in a typical Job Duty Task Analysis (JDTA) but is often discarded as instructional content 
is developed. 


6.1 Next Steps 
In FY2019, the ADL Initiative will pursue multiple activities related to CBL and maturing the overall TLA. 


1.) The ADL Initiative has partnered with Air Education and Training Command (AETC) and 
USALearning to conduct a research effort (Air Force Learning Services Ecosystem (AFLSE)/TLA 
Baseline project). This project supports the inclusion of CBL and other TLA technologies into the 
AFLSE. A competency framework is being created for an Air Force Specialty Code (AFSC), TLA 
components will be integrated into AFLSE systems, and evidence will be collected from 
authoritative sources over a 6-month period. 
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2.) The ADL Initiative has prioritized the FY19 topic areas for the Broad Agency Announcement 
(BAA) to include a focus on defining the technical underpinnings of competencies, establishing a 
robust metadata strategy, and understanding the interplay between learner profile data, authoritative 
sources, and services. 


3.) The ADL Initiative staff is continuing work on the 2019 TLA reference implementation and its 
migration to a data streaming architecture that continues research into federated LRSs and 
associated business rules, the implementation of a competency DAG, and the establishment of a 
modern learner profile. 


4.) The Competency and Skills System (CaSS) is being hardened and stabilized for use in the 
AFLSE/TLA baseline project and potential transition to the USALearning/ADL Initiative learning 
warehouse. Authoring tools are being matured to better evaluate the associated level of effort for 
defining competencies, creating frameworks, and aligning evidence repositories. 


5.) The ADL Initiative and USALearning are working through the Combatant Commanders Exercise 
Engagement and Training Transformation (CE2T2) program to take an enterprise look at the 
numerous DoD policies that need to change to support CLB within the DoD. This project is also 
collecting requirements across the services to enable a universal DoD learner profile and is 
exploring the concept of federated governance for authoring and maintaining competency 
frameworks across the numerous domains associated with the DoD. 


6.2 Conclusions and Recommendations 


CBL is the bridge to having a trusted and informed prediction of how service members can perform in their 
jobs. This data affords a new view of the warfighter in general and enables the DoD to better prepare its 
service members for any potential mission. It has the potential to align the different organizations across 
the DoD responsible for training and education, human resources/personnel, and the operational work 
environment. This alignment brings the potential for increased readiness and a long-term cost savings by 
optimizing the way our workforce does, trains, practices, and executes their jobs. 


The migration to CBL provides strategic insights into overall human performance within a JCS. As 
individual competencies are evaluated, trends can be identified that expose faults in organizational 
workflow, system usability, and training & education. A big challenge with current systems is the 
disconnected pools of data stored across a learner’s lifetime and across institutional boundaries. Those 
disconnected resources hold great potential but are currently barriers to realizing the insights that CBL can 
provide to workforce planning and talent management, mission effectiveness, and force readiness. 


While the technology challenges associated with the migration to CBL are not trivial, the change to culture, 
policies, and budgets is huge. The underlying concepts of CBL are complex, difficult to describe, and 
require time to manifest a return on investment. Access and permissions need to be granted to pull 
evidentiary data from operational systems, and acquisition strategies need to be revised to align with an 
overall DoD data strategy that links performance outcomes with mission readiness to enable future planning 
and decision making. 


With the different learning modernization programs going on across the DoD, we are afforded the 
opportunity to lay the groundwork for intelligent CBL systems, federated data governance, and data 
portability that is required to maintain our global security posture. 
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APPENDIX A - CONCEPTS AND DEFINITIONS 


Assertions: This is a foundational concept to Competency Based Learning. Assertions are defined by 
Webster’s Dictionary as a “statements of fact or belief.” In terms of competencies, an assertion is the 
expression of a trusted fact or belief about a learner’s proficiency in specific KSAOs associated with a 
competency. Assertions are typically proffered with corresponding evidence that infers a specific 
competency for a specific learner. Ideally, assertion evidence is immutable and persistent so that it cannot 
be repudiated. This evidence should be interoperable between learning ecosystems to allow different levels 
of sharing and mutual recognition of competency data across domains (e.g., military and civilian). This is 
exemplified in the relationship between the Defense Health Agency (DHA) and the Medical Departments 
within each service. This is also demonstrated in the relationships between the different elements of the 
Intelligence Community (IC) where teams frequently come together to collaborate. 


Assessment: Assessment is the collection and evaluation of evidence that can be used to make assertions 
of competency. Many learning activities (e.g., reading a book) contribute to an individual’s proficiency in 
a specific competency, but formal assessments are considered to have a higher confidence value in terms 
of evidence. In The T&E world, these may come from capstone training events or any number of formative 
and summative testing events. In the operational world, these come in the form of job reviews, after-action 
reviews, or other trusted, authoritative source. Thus, assessments are the driver for the evidence behind an 
individual’s proficiency within a given competency and in a given context. 


The context is critical for translating relevance from the different perspectives between a pure learning 
environment and an actual work environment. Context helps define proficiency levels for different models 
of skill acquisition (e.g., Dreyfus and Dreyfus'’). Adjudication of the assessment data within the overall 
competency framework is maintained by a competency management system and is captured in an assertion. 
In the current paradigm, institutions provide a credential (e.g., degree) that rolls up all assessments, but 
loses details of learners’ experiences. Competency Management Systems could take in evidence from 
workplace efforts of individuals to maintain proficiency ratings according to accreditation standards. 


Competency Based Learning (CBL): Competency Based Learning is focused on the application of skills 
and knowledge rather than on recollections of abstract concepts'*. Mastery is demonstrated through 
performance where learners apply the required KSAOs they need to do their job. The current model of 
education within the DoD places trust in the formal training each service member receives with little data 
collected about learning that occurs on the job. Competencies must be defined for all the different jobs and 
roles performed. These must be tied into a framework that accommodates technical skills, as well as 
professional, behavioral, functional, or even organizational skills. Key performance indicators for each 
competency should be identified and tied to mission outcomes and the systems/components used to achieve 
those outcomes. For example, aviation maintenance records often identify what tasks are being performed 
and by who. Using this information, a great deal of evidence is presented on the aviation mechanic’s level 
of technical competencies by providing a ledger of the tasks they’ ve performed. 


'! Dreyfus, Stuart E.; Dreyfus, Hubert L. (February 1980). "A Five-Stage Model of the Mental Activities Involved in 
Directed Skill Acquisition" 

2 Gervais, J. (2016). "The operational definition of competency-based education". The Journal of Competency-Based 
Education. 1: 98-106. https://onlinelibrary.wiley.com/doi/full/10.1002/cbe2.1011 
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Competencies are defined as networks of KSAOs that collectively define a set of related performance 
elements, along with any collected evidence that the learner has achieved mastery of these elements at some 
gradated level of performance (journeyman, master, etc.). A competency framework embodies this network 
and establishes the relationships between those elements. In traditional education, competencies are pursued 
by each learner according to a prescribed curriculum. This is both culturally driven and necessary from a 
practical, resource-management perspective. As seen in planning for its Ready Relevant Learning (RRL) 
program!, the US Navy is migrating a Sailor’s initial training phases into a shortened format, then 
developing new instructional-design techniques to shift the remaining learning requirements to on-the-job 
training with performance suppott aides. 


Competency: The DoD defines a competency as “an observable measurable pattern of knowledge, abilities, 
skills, and other characteristics that individuals need to successfully perform their work.”'* This conceptual 
definition does not specify the supporting characteristics that makes competencies actionable across a 
learning ecosystem, like an implementation of TLA. To that end, the ADL Initiative proposes an expansion 
on the previous definition: competencies are trusted, discrete, and encapsulated representations of KSAOs 
and other characteristics at the individual level necessary for performing a role within a specific context. 


Competencies can be assessed and are severable under a specific context. Severability varies depending on 
how the KSAO is trusted and demonstrated within a network of competencies. Competencies must be 
functional on a global scale to evolve with learners over time. For example, soft-skills, like time- 
management, are critical during individuals’ development and subsequent careers, but rarely are a 
component of their formal records. A model of how competencies relate as a network is a requirement for 
maximizing the return of a competency service within a TLA implementation. 


Competency Object: An umbrella term used to describe any abstract statement of learning or performance 
expectations, and information related to the statement. Statements can be learning outcomes, competencies 
per se, learning objectives, professional roles, topics, classifications/collections, etc. A competency object 
might include the following: 


e A unique identifier, competency name, and detailed information 

e Descriptions of required activities and behaviors (pre-requisites, physical requirements, reading levels, 
ethics, etc.) 

Associated Knowledge, Skills, Attitudes, and Other facets (KSAOs) 

Enabling Learning Objectives and Terminal Learning Objectives (ELOs/TLOs) 

Tasks, conditions, and standards 

Assessments, certifications, qualifications 


Competency Framework: A competency framework is a collection of competencies that provide 
clarification about successful performance in different roles at different levels throughout the organization. 
Competency frameworks are a network of KSAOs (or other characteristics) that can have many non- 
exclusive relationships. These frameworks can be associated with other constructs outside of a required 
learning outcome. A competency framework organizes the KSAOs needed to perform successfully in a 
work setting, such as a role, occupation, or industry, or within a pure learning setting such as an academic 


3 "Ready Relevant Learning (RRL) Official Website." U.S. Navy Hosting. Accessed May 14, 2019. 
https://www. public.navy.mil/usff/rrl/Pages/default.aspx. 

'4 DoD Competency Definition, DoDI 1400.25, June 2017 
http://www.esd.whs.mil/Portals/54/Documents/DD/issuances/140025/140025_vol250.pdf 
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or vocational-training institution. The competency framework can be used as a resource for developing 
curriculum and selecting training materials, identifying licensure and certification requirements, writing 
role descriptions, recruiting and hiring workers, and evaluating employee performance. 


The creation of competency frameworks requires extensive planning and investment. There is no 
universally consistent way for creating a competency framework. Many existing frameworks are 
represented by text that is scanned as Portable Document Format (pdf) files. They tend to not be easily 
searched, are not syntactically understandable to most computers, and do not have any capacity for 
automated configuration management as the frameworks evolve over time. A complete framework should 
include relationships to potential or allowable evidence (in the form of experiences or assessments from the 
work environment) that correlate to a demonstration of achievement for each competency by providing 
trusted evidence of mastery. 


Competency Management System: A competency management system provides the foundation to 
strategic talent-management practices such as workforce planning, acquiring top talent, and developing 
employees to optimize their strengths. The ability to preserve granular data on KSAOs and maintain 
traceability between captured evidence and the state of an individual offers significant improvement over 
the current credential-oriented practices in talent management. Effective and automated competency 
management can create a real-time and predictive inventory of the capability of any workforce. Competency 
management is expected to provide these key capabilities: 


e Enriched understanding of expected behaviors and performance. The quickest path to improving 
performance starts by knowing the target performance. 


e Improved talent management. The mastery estimates about competence inform leadership about 
current and future capability. Data and analytics about employees’ skills and knowledge are used by 
the competency management system as evidence to predict or assert confidence about a specific 
competency object. 


e Optimized talent development. Enables the process of establishing training goals and plans that link 
to individual goal attainment, career planning, and succession planning. It also facilitates a personalized 
learning plan by aligning content with competencies specific to an individual’s role within the 
organization. 


e Enhanced _ talent pipeline. Competency management enables on-demand information about an 
individual’s competency mastery and readiness to move into next-level or other critical roles. An 
individual’s Career Trajectory is derived from their prior experience, education, and expertise and is 


influenced by numerous external and internal components. 


e Improved operational efficiency. Competency management facilitates business-driven learning and 
development, eliminates non-value-added training, highlights strengths to be further developed, flags 
critical skill gaps for mitigation, and generates higher levels of employee and leader satisfaction with 
their overall experience with the organization. 


e Integrated broker. There are numerous approaches to building competency frameworks. 
Competencies are encapsulated differently across industries, geographic regions, training domains, and 
even across our military services. 
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A Competency Management System preserves all KSAO data within a framework and aligns competencies 
with organizational goals, drives experience-building opportunities for individuals within the organization, 
and acts as a linkage between competencies and other talent-development processes. By preserving data 
throughout a learner’s career, this system can act as a brokerage service between a layer of trusted data that 
encapsulates the relationship between human performance and mission outcomes. Figure 11 depicts the 
roll-up of KSAO data. 
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Figure 11. KSAO Roll-Up Competency - The evidence used to infer competency is derived from authoritative 
sources aligned with the KSAOs in any number of competency frameworks and preserved for further analysis and/or 
proof. 


Competency vs. Credential: A network of competencies typically has varying mastery levels as part of its 
credentialing model. Previous levels contribute to the next level of mastery, and competency elements 
within the various levels may atrophy over time from disuse. Competencies are valued by the trust 
associated with them. Legacy credentialing models are dependent on the social value of groups of people 
trusting each other. The future of digital competencies derives trust from immutable data associated with 
discrete evidence that cannot be repudiated. These data can build from legacy constructs like certificates 
and degrees but preserving the raw elements of individual experiences for persistent consideration changes 
the possibilities for future education and training. 


Credentials: A trusted exchange format that encapsulates a single competency or a set of competencies 
into an artifact from an accrediting organization. A credential is issued by an entity with authoritative power 
and provides proof of an individual’s qualification or competence in a subject. These artifacts range widely 
from a college degree to a professional certificate to a badge or a micro-credential. Possessing a credential 
not only helps one to prove competency and capability within a field, it also serves as verification that the 
individual is properly trained and equipped to carry out their duties within their specific vocations or 
disciplines. 
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In the current paradigm, credentials don’t typically include data about the curricula or experience associated 
with the award of the credential. This is seen in degrees, where many years of abstract study and practical 
demonstration are distilled into a degree, associated with a transcript, and tied to a grade point average. For 
example, the domain of electrical engineering, governed by Accreditation Board for Engineering and 
Technology (ABET), Inc.,'° has an extensive requirement list for what an institution must do to have its 
bachelor’s degree program accredited. At the point of conferring a degree, the student becomes a 
professional, and most of the performance data collected in obtaining the degree is lost. The degree, 
transcript, and GPA are trusted, but they do not provide much detail on the performance capabilities of the 
graduates. In theory, credentials can be issued at any level of granularity in relation to a competency and 
may “roll up” in many cases. Examples include assignments “rolling up” to transcript recorded grades, 
stacking professional certifications, or apprenticeships. Types of credentials include: 


e Badges: Represent a way of acknowledging achievements or skill acquisition in a granular fashion. 
A badge is designed to motivate behavior, recognize achievement, and establish credibility. Badges 
capture knowledge, skills, and accomplishments that help transfer learning across different 
communities. Digital badges have emerged to document knowledge about a subject, professional 
development, leadership, or other accomplishments. The Open Badge’® initiative and other 
standards are built on the concept of capturing accomplishments, relating them to a graphical 
depiction, and then providing a validation process for assigning them. They do not necessarily need 
to represent attainment of a unique competency but have value in terms of social learning and 
motivation. 


e Certification: A formal, but typically voluntary process that recognizes and validates an 
individual’s qualifications in a certain subject. Certification is earned by individuals to assure they 
are qualified to perform a role or task through the acknowledgement of developmental 
achievement. Typically, a non-governmental entity grants a time-limited recognition to an 
individual after verifying that they have met a pre-determined criterion. Certificates verify that a 
professional has achieved a specific level of competence in a specific subject area and assures 
employers that the individual can handle the challenges associated with the role. Certifications are 
typically earned from a professional society or a commercial entity, like Microsoft or Cisco, and 
must be renewed periodically, generally through continuing education units and/or a reassessment 
of qualifications/capabilities. Examples include a Certified Project Manager from the Project 
Management Institute'’, or a Microsoft Certified Solutions Expert (MCSE)'*. Unlike licensure, 
individuals do not typically need to be certified to engage in given occupations. Sometimes, 
however, certifications become so important to certification attainment that they are quasi- 
mandatory. 


e Licensure: A mandatory process by which a governmental agency grants time-limited permission 
to an individual to engage in a given occupation after verifying that he or she has met pre- 


‘ABET Engineering program requirements http://www.abet.org/accreditation/accreditation-criteria/criteria-for- 
accrediting-engineering-programs-2017-2018/ 
‘6 https://www.imsglobal.org/sites/default/files/Badges/OB v2p0Final/index.html#intro-example 


17 www.pmi.org 
'8 https://www.microsoft.com/en-us/learning/mcse-certification.aspx 
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determined and standardized criteria. Examples include registered nurses”, a real estate agent”, or 
professional engineers”'. The goal of licensure is to ensure that licensees possess enough levels of 
competency to ensure that public health, safety, and welfare are reasonably well-protected. 


Micro-Learning and Micro-Credentials: Like badges, micro-credentials are performance-based 
assessments that allow learners to showcase their growing skills. Micro-credential is a term used 
to reference a subset of competencies that is significant enough to report (such as clearance to 
perform a medical procedure with a new device) that can be unique or build towards a more 
substantial credential. Micro-credentials are typically on-demand, shareable, and personalized. In 
some professional communities, individuals collect micro-credentials to document continuing 
education units that apply towards re-certification. To put micro-credentials in a broader context, 
learners typically have a choice in what credentials they want to pursue and can create their own 
developmental plans. 


Credential Management System: A credential management system captures performance of individuals 
and publishes a certificate, degree, or qualification that is trusted in an organization. In the civilian domain, 


independent accrediting bodies play a critical role. In the military, independent accrediting bodies affect 


some domains but are not required across all areas. Credentials as the key measurement artifact tend to 


generalize people into designations that obfuscate granular metrics on their performance. This section 


describes how measurements become obscured in credentialing systems, and establishes granularity and 
traceability related to individual knowledge and skill as key concerns addressed by CBL. 
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Figure 12. Credential Roll-Up - The services all have varying methods to prove proficiency upon being assigned a 
role in an operational environment. However, this is often based on assumptions about typically non-expiring 
credentials. By removing granular data, specific grading, and not providing a traceable data-set, leadership at the 
working level must build their own internal model of what subordinates can do based on limited data. 


'9 https://www.ncsbn.org/nclex.htm 


20 https://www.kapre.com/resources/real-estate/requirements-to-get-a-real-estate-license/ 
p p q £ 


71 https://ncees.org/engineering/pe/ 
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The Armed Services Vocational Aptitude Battery (ASVAB) combined with a High School Diploma or 
equivalent degree provides the baseline measurement for recruitment. After recruitment these few 
measurements represent the total measure of an individual entering the accession training. Compared to the 
individual’s previous data set captured in high school, college, work environments, and other sources, the 
granularity of records has gone to a scale that obfuscates what the individual is capable of beyond anecdotal 
discussion. Specifically, demonstration of knowledge or skill upon entering the military requires self- 
identification on the individual and proving their ability in the workplace. Traceability is also sacrificed. 
An example is that aptitude to perform complex math problems identified by the ASVAB does not identify 
what formal mathematics education a service member obtained prior to enlistment. Many military 
schoolhouses pursue additional tutoring services for service members who join with difficulty reading or 
writing. If those records were preserved, overall learning could be tracked, and the DoD could better model 
how to educate moving forward at the individual level. 


Upon completion of training, records of individual performance are captured within a credential. Individual 
performance, areas of strength or weakness, and soft-skill application are all lost, even though staff at 
training commands take note of these factors either formally or informally. As a service member enters the 
career phase, work experiences, as well as training and education experiences foster knowledge and skills 
to enable capture of credentials. Post accession service members are represented by their credentials, rank, 
and current medical condition at the talent management level. As shown in Figure 12, “Fit and Fill” is a 
common euphemism in the DoD for finding the right combination of rank and Navy Enlisted 
Classification/Military Occupational Specialty (NEC/MOS) credentials then placing them in a role. 


Learner Profile: Learner profiles have the potential to empower personalized learning through the 
accumulation of better data that can inform learning in new and meaningful ways. A learner profile typically 
includes a broad range of data, including demographic data, data about student interests, learning 
preferences, inter- and intra-personal skills, existing competencies, and additional competencies that still 
need to be developed (in the personal, social-emotional, academic and career arenas). It might also include 
information on credentials or other student-learning strengths, needs, and types of support that have been 
successful in the past. 


A learner profile might contain explicit goals about desired learner outcomes. When coupled with behaviors 
and experiences of other students, however, inferred data may inform valuable feedback sessions addressing 
fragile or missing skills, enable decisions about student placement, or highlight specific interventions 
needed to support success. Some elements of learner profiles afford learners a great deal of control. Just as 
learner profiles evolve over time to support personalization at the learner level, it is anticipated the learner 
has access to obtain, share, and interact with these artifacts, as well as controlling access to them by others, 
including other people, organizations, applications, and even other TLA components. 


Safeguarding learner data to preserve privacy is an important legal and ethical consideration. This is 
especially true in the future learning ecosystem, where data-rich learner profiles and student achievement 
data are collected, stored and shared. The evolution from learner profile to student-owned pathways is key. 


In education, a transcript is a copy of a student’s permanent academic record, typically including all courses 
taken, all grades received, all honors received, and all degrees conferred to a student. A learner profile needs 
to consider this formal information while also being dynamic enough to document other achievements as 
individual learners are constantly changing and will become competent in new areas over time. This 
information about learners should be used — with learners’ full cooperation — to determine their unique 
paths to achieving proficiency in all required and desired competencies throughout life. 
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Experience Application Program Interface (xAPI): xAPI” is a specification currently going through the 
Institute of Electrical and Electronics Engineers (IEEE) - Learning Technology Standards Committee 


(LTSC) standards-development process (https://www.tagxapi.org/). It facilitates the collection of data 


about the wide range of experiences a person has (both online and offline). xAPI captures data in a 
consistent format about a person’s or group’s activities from many technologies. Different systems can 
securely communicate by capturing and sharing a stream of activities using xAPI simple vocabulary. xAPI 
specifies a structure to describe learning experiences and defines how these descriptions can be exchanged 
electronically. The main components of xAPI are the data structure called statements and the data 
storage/retrieval capability called the LRS. The xAPI specification has stringent requirements on the 
structure of this data and the capabilities of the LRS. Statements employ an actor, a verb, and an object to 
describe any experience. Each statement also includes timestamps and unique and traceable identifiers. 


xAPI Profile: The “Companion Specification for xAPI Vocabularies””* released in March 2016, describes 
the general approach needed to strengthen the semantic interoperability of xAPI. The “xAPI Profiles 
Specification” refines and extends the vocabulary specification to offer a common way to express 
controlled vocabularies, provide instructions on the formation of xAPI statements, and describe patterns of 
xAPI usage that provide additional context to a domain, device, or system. The xAPI Profiles Specification 
also adds tools to support authoring, management, discovery and/or adoption, including additional data 
elements and properties needed to support numerous use cases. A Benefit of xAPI Profiles is that they can 


standardize xAPI implementations across the “Internet of Learning Things (IoLT).” 


22x API Specification https://github.com/adInet/xAPI-Spec) 
3 https://adl. gitbooks.io/companion-specification-for-xapi-vocabularies/content/ 
4 https://github.com/adInet/xapi-profiles 
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APPENDIX B — TECHNICAL STANDARDS AND SPECIFICATIONS 


Achievement Standards Network™ (ASN™): ASN™ provides access to machine-readable 


representations of learning objectives and curriculum standards. It provides a Resource Description 
Framework (RDF)-based framework based on the Dublin Core Metadata Initiative’s (DCMI) syntax- 
independent abstract information model (DCAM). The ASN™ framework is made up of Standards 
Documents and Statements. The Standards Document represents the overall competency framework, while 
Statements represent the individual achievements within the overall framework that can be asserted. A set 
of core properties define the relationships between the two in terms of an Entity-Relationship model. 
Structural relationships replicate the relationships between different components of the Standards 
Document, and semantic relationships define the relationships of meaning between statements (e.g., 
assertion of equivalence). 


ASN™ is designed for interoperability and open access to learning objectives. It has seen wide adoption in 
the K-12 community. The ASN Description Framework (ASN-DF) provides the means to create ASN™ 
profiles through inclusion of additional properties/classes from other namespaces. ASN-DF provides a 
small vocabulary of classes and properties and a set of mechanisms for tailoring an ASN™ Profile based 
on the Dublin Core's conceptualization of application profiles and description templates. 


Competencies and Academic Standards Exchange™ specification (CASE™): The IMS Global 
Competencies created the CASE™ specification to define how systems exchange and manage information 
about learning standards and/or competencies in a consistent and digitally-referenceable way. CASE™ 
connects standards and competencies to performance criteria and provides a way to transmit rubrics 
between various platforms. The CASE™ Service specification is the definition of how systems achieve the 
exchange of information about learning standards and/or competencies. Within CASE™, the underlying 
structure of both a competency and an academic standard are represented using the same data model. The 
data model is composed of three core constructs: 


e The Root Definition Document — the top-level structure that collects the set of statements that 
define the individual competencies/academic standards. Within CASE, this structure is 
identified as the CFDocument. 


e The Set of Composition Statements — the set of statements into which the top-level 
competency/academic standards have been decomposed. Within CASE, this structure is 
identified as the CFItem. 


e The Rubric — the detailed definition of how it can be determined that mastery of the associated 
competency/standard has been achieved. This requires the definition of specific criteria used 
for each of the scores that can be awarded during an assessment. Within CASE, this structure 
is identified as the CFRubric. 


The CASE™ specification defines internal relationships between statements like Parent/Child, Precedes, Is 
Related To, Exemplar, and Is Part Of. All competency frameworks published in CASE™ format can be 
linked together in a network of equivalent or aligned competencies. Having universal identifiers for each 
competency makes it possible for any tools or applications to share information between systems. The 
composition and logical structure of the data model for this hierarchical structure uses a linked list approach. 
Competencies must be related to programs, courses and modules, assessments, course materials, and 
learners’ records of achievement. The CASE™ rubric is also suited for establishing the criteria and 
performance levels for earning credentials. 
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Occupational Information Network (O*Net): Valid data is essential to understanding the rapidly 
changing nature of work and how it affects the workforce. O*NET was developed under the sponsorship 


of the U.S. Department of Labor/Employment and Training Administration (USDOL/ETA) to facilitate and 
maintain a skilled workforce. Central to the project is the O*NET database, containing hundreds of 
standardized and occupation-specific descriptors on almost 1,000 occupations covering the entire U.S. 
economy. 


Every occupation requires a different mix of knowledge, skills, and abilities and is performed using a variety 
of activities and tasks. As shown below in Figure 13, the O*NET database identifies, defines, describes, 
and classifies occupations through Experience Requirements (Training, Licensing), Work Requirements 
(Basic and Functional Skills), Occupation Requirements (Activities, Context), Worker Characteristics, 
Occupation Specific Information, and Occupational Characteristics. The O*NET Database includes a 
Military Transition search that connects occupations in the O*NET database to other classifications systems 
within the Military. This is accomplished through data from the Military Occupational Classification 
(MOC) system at the Defense Manpower Data Center (DMDC). This capability is also available via web 
services*>. Other resources include “Careers in the Military” and links to Army, Navy, Marine Corps, and 
Air Force COOL projects. Google”® has recently released their own career recommender that leverages 
O*NET and military classification data. 
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Figure 13. O*NET Content Model — This model provides the framework that identifies and organizes the 
distinguishing characteristics of an occupation. The model defines the key features of an occupation as a standardized, 
measurable set of variables called descriptors. This hierarchical model starts with six domains that expand to 277 
descriptors. 


°5 https://www.onetcenter.org/crosswalks.html 
6 Google Veteran Job Finder https://qz.com/work/137133 1/a-simple-tweak-to-google-search-now-helps-veterans-to- 
find-jobs/ 


27 


MedBiquitous Competency Framework (MEDBIQ CF): The use of outcome and competency 


frameworks is a growing part of healthcare education and maintenance of certification. The MedBiquitous 
Competency Framework, ANSI /MEDBIQ CF.10.1-2012, is a technical standard for representing 
competency frameworks in eXtensible Markup Language (XML). MedBiquitous is accredited by the 
American National Standards Institute (ANSI). Organizations that publish competency frameworks can do 
so in this standard format, making it easier to integrate competency frameworks into educational 
technologies like curriculum management systems. 


The data model establishes relationships between competency objects that narrow or broaden the definition 
of overall competency. The MedBiquitous Competency Object specification provides a consistent format 
and data structure for defining a competency object, an abstract statement of learning or performance 
expectations, and information related to the statement. Statements can be learning outcomes, competencies, 
learning objectives, professional roles, topics, classifications/collections, etc. The Competency Object may 
include additional data to expand on or support the statement. This specification is meant to be used in 
concert with complementary specifications, including the MedBiquitous Competency Framework 
specification. 


Reusable Competency Definition (RDC): The IEEE Standard for Learning Technology—Data Model for 
Reusable Competency Definitions (RCD) WG 20 will revise the 2008 Reusable Competency Definition 
(RCD) (1484.20.1) standard. The current standard defines a data model for describing, referencing, and 
sharing competency definitions, primarily in the context of online and distributed learning. The standard 
provides a way to represent formally the key characteristics of a competency, independent of its use in any 
specific context. It enables interoperability among learning systems that deal with competency information 
by providing a means for them to refer to common definitions with common meanings. The competencies 
WG 20 intends to take the Credential Ecosystem Mapping Project’s mapping of competencies metadata 
and update RCD to represent the most common elements that are found in multiple standards addressing 
competencies and competency frameworks. This effort started in September 2018 and will be monitored 
for progress. 
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basic grammatical construct in making CTDL assertions about things and is comprised of three simple 
components: a subject, a predicate and an object. The comprehensiveness and scope of this specification 
makes it ideal for defining TLA Credentials. The CTDL is comprised of a small set of primary classes 
identifying major entities in the credentialing ecosystem. 


As shown in Figure 14, CTDL provides clear guidance about the structure and terms used to describe 
attributes and assert relationships between all classes for each credential. Two super classes called Agent 
and Credential define families or sets of subclasses used throughout the CTDL. The primary classes also 
include the ConditionProfile used to define sets of constraints on the credential described by an 
Assessment Profile, Learning Opportunity Profile, or Competency Framework. These are used to express 
learning goals and outcomes in terms of knowledge, skills, and abilities across a diverse set of profiles. 


Credential Registry: The Credential Registry”* is both a repository of information regarding credentials 
and a set of services that make it easier to use that information. Together with CTDL, it provides for the 
capturing, connecting, archiving, and sharing of information about credentials (including degrees, 
certificates, certifications, licenses, apprenticeships, badges, micro-credentials, and more). It also registers 
credentialing organizations that issue these credentials to individual credential holders, including 
universities, colleges, schools, industry and professional associations, certification organizations, military, 
and more. 


Credential Registry also allows users to see what various credentials represent in terms of competencies, 
transfer value, assessment rigor, third-party approval status, and much more. The Credential Registry works 
with CTDL to allow the storage of credentials that have been expressed using that specification. The 
Credential Engine project’s developers are using the Dublin Core Application Profiles process to create 
systems that communicate virtually all aspects of credentials. The Technical Advisory Committee (TAC) 
promotes collaboration across, and harmonization of, standardization initiatives that are developing data 
models, vocabularies, and schemas for credentials and competency frameworks, as well as related 
competency information such as criticality ratings and assessment data typically captured with a wide 
variety of systems. The Credential Registry uses technology and CTDL to capture, link, update, and share 
up-to-date information about credentials so it can be organized and centralized within the Registry, and 
then made searchable by customized applications, and linkable from anywhere on the open Web. 


The Credential Registry serves as acomplement to the CTDL specification and meets required TLA registry 
capabilities for defining credentials. It is important to note that this is only a registry and, as such, only 
stores the description of a credential in the abstract sense. From this perspective, the registry can be used to 
house credentials created with other specifications (e.g., CASE™). The Credential Registry is not a store 
of personal information or of personally-obtained credentials. That information will be stored in the TLA 
Learner Profile. 


IMS Global Open Badges: Open Badges” are information-rich visual representations of verifiable 
achievements earned by recipients. Open Badges is a technical specification and set of associated open- 
source software designed to enable the creation of verifiable credentials across a broad spectrum of learning 
experiences. The Open Badges standard describes a method for packaging information about 


28 https://www.credentialengine.org/credentialregistry 
° https://www.imsglobal.org/activity/digital-credentials-and-badges 
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accomplishments, embedding it into portable image files as digital badges, and establishing resources for 
its validation and verification. Open Badges contain detailed metadata about achievements, such as who 
earned a badge, who issued it, and what it means. The Open Badges 2.0 specification expands on this 
capability to include versioning, endorsements, and full use of JavaScript Object Notation Linked Data 
(JSON-LD)*°. 


The Open Badges Vocabulary defines several data classes used to express achievements that are 
understandable within software and services that implement Open Badges. There are three core data classes: 
Assertions, Badge Classes, and Profiles. Each data class is a collection of properties and values, and each 
defines which are mandatory and optional, as well as the restrictions on the values those properties may 
take. Each Badge Object may have additional properties in the form of an Open Badges Extension, a 
structure that follows a standard format so that any issuer, earner, or consumer can understand the 
information added to badges. Assertions are representations of an awarded badge, packaged for 
transmission as JSON objects with a set of mandatory and optional properties. 


Open Badges are expressed as linked data so that badge resources can be connected and references as a 
permanent fixture in a TLA Learner Profile. The attributes defined by the Badge Class and Issuer Profile 
are clear and the rules surrounding the specification drive the interoperability. Finally, the URL usage of 
Badges and many of their properties (Evidence, Criteria, Image, Issuer, etc.) enable them to be easily 
accessed on the Web. 


W3C Verifiable Claims: The mission of the Verifiable Claims Working Group (VCWG) is to make 
expressing and exchanging credentials that have been verified by a third party easier and more secure on 
the Web. Driver’s licenses are used to claim that we can operate a motor vehicle, university degrees can be 
used to claim our education status, and government-issued passports enable holders to travel between 
countries. This specification provides a standard way to express these sorts of claims on the Web in a way 
that is cryptographically secure, privacy respecting, and automatically verifiable. A verifiable claim is a 
claim that is effectively tamper-proof, and whose authorship can be cryptographically verified. Multiple 
claims may be bundled together into a set of claims. 


The basic components of a set of verifiable claims include a Subject Identifier, Claims about the Subject, 
Claim Set Metadata, and a Digital Signature. Both the Entity Profile Model and Entity Credential Model 
consist of a collection of name-value pairs that are referred to as properties in the data model. The link 
between the two is in the ID property. The Entity Profile Model defines a subject identifier in the id 
property, while the claims section of the Entity Credential Model uses the id property to refer to that subject 
identifier. Unlike the properties in the Entity Profile Model, the properties in the claim section of the Entity 
Credential Model are claims made by an entity about the subject defined in an entity profile. 


Additional research is required to better evaluate its applicability to the TLA. However, the defined data 
model and structure of this specification implies that it could connect to the TLA both semantically and 
syntactically. The data associated with verifiable claims are largely susceptible to privacy violations when 
shared with other TLA components, so many of the Verifiable Claims — Use Cases should be examined to 
inform how Personal Identifying Information (PII) connected to credentials can be protected within the 
security boundaries of the TLA. 


3° JSON-LD https://json-ld.org/ 
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Comprehensive Learner Record (CLR): IMS Global led the development of the CLR 


(https://www.imsglobal.org/activity/comprehensive-learner-record), formerly known as the Extended 
Transcript, it is designed to support traditional academic programs and co-curricular and competency-based 


education in order to capture and communicate a learner's achievements in verifiable digital form. The 
vision for the learner record is transformative in its potential, beyond providing relevant student 
competencies and skills. Closely following the developing registrar guidance of American Association of 
Collegiate Registrars and Admissions Officers (AACRAO), the vision for CLR is a secure, student-centered 
digital record for the 21st century. 


The CLR standard is a new generation of secure, verifiable digital records for learners, containing all nature 
of learning experiences and achievements including courses, competencies, skills, co-curricular 
achievements, prior learning, internships and experiential learning. This variety of credentials targeted by 
this standard could make it viable as a TLA Credential standard. The CLR seems to be versioned according 
to the Extended Transcript standard. At the time of this report, the support of these efforts, and the emerging 
IMS Pathways specification are unknown. 
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